The terms "virtual environment," "virtual world," and "virtual reality" are used interchangeably to describe a computer-simulated environment (intended to be immersive) which includes a graphic display (from a user's first person perspective, in a form intended to be immersive to the user), and optionally also sounds which simulate environmental sounds. The abbreviation "VR" will sometimes be used herein to denote "virtual reality," "virtual environment," or "virtual world". A computer system programmed with software, and including peripheral devices, for producing a virtual environment will sometimes be referred to herein as a VR system or VR processor.
The graphic display generated by a VR system can be a two-dimensional (2D) or a three-dimensional (3D) display. Typically, a VR system includes an input device and user interface software which enable a user to interact with the scene being displayed, typically to simulate motion in the virtual environment or manipulation of displayed representations of objects ("virtual objects") in the virtual environment. Typically, the illusion of immersion in a virtual reality system is strengthened by the use of head-tracking or some other such system which directs the computer to generate images along the area of viewing interest of the user.
The present invention is a method and apparatus particularly useful for creating and/or controlling virtual environments. A VR system which embodies the invention can rapidly and inexpensively create, animate, or otherwise control a wide variety of entertaining virtual environments and virtual objects in response to music or in response to prerecorded "control tracks" which correspond to audio signals (such as music).
While currently being used in the research and scientific communities, VR systems are becoming less expensive and are poised to reach the consumer electronics market as entertainment devices.
VR systems must generate a much greater amount of content data (image data and audio data simulating environmental appearance and sounds) than must be generated in most other electronic media. Whereas video game systems require complex scenes to be generated and themes to be programmed, such systems can easily limit the scope of the game content because they can easily constrain the player to move in a few simple directions (e.g., left and right) and need only produce images to be presented on flat screen monitors or on simple 3D field-sequential type monitors.
In contrast, by their very nature, VR systems allow the user to look around and fly around in many different directions and positions. Even where the user is constrained to look only toward the left or the right, VR systems must construct complete representations of 3D worlds. This complexity has made it very difficult to generate virtual worlds for the consumer entertainment market in a quick fashion.
In addition to the complexity of creating static 3D models for virtual worlds, it has also been difficult to control the dynamics of virtual worlds. VR systems to date are notorious for providing only very boring and nearly static environments. The few VR systems that include dynamic motions of the virtual world either base such motions on physical laws (such as gravity) or base the motions on corresponding motions produced by human users (such as motion of the fingers of a user wearing a conventional "glove" input device).
The present invention overcomes the limitations of conventional VR systems by providing an efficient way to generate content data (i.e., animated image data and audio data) to fill or populate a virtual environment in a choreographed response to input music signals.
There has long been an interest in the virtual reality field with respect to the possibility of virtual musical instruments and the creation of new and novel instruments within a virtual world. The present invention is a radical shift from previous attempts to combine music and virtual environments.
Conventional efforts to integrate music with virtual environments have, to date, all been directed toward creation of music from a virtual environment. The musical expression of the user has been treated as an urge seeking to be brought forth, and virtual environments have been seen as vehicles for the user to perform music or dance without having to learn special physical skills. Much effort has been made to make sounds appear to be coming from virtual objects in the virtual environment. This has been done by running audio into the VR system and then convolving the audio in such a way as to make it appear to come from a certain place in the virtual environment.
For example, at the NASA Ames View Lab, Scott Fisher, Rick Jacoby, and others explored virtual environments. One aspect of the research was the integration of audio into the virtual experience. This included the use of audio cues for such purposes as telling one if one bumped into a virtual object, but there was no tactile feedback for such events. The research pushed into the more artistic realm of creation of music in the context of a virtual world.
Mark Bolas and Phil Stone created the Virtual Theremin and virtual drum kit. In this system, the user wore a glove and a hand tracker and moved the gloved hand to manipulate virtual objects which were in turn linked to various synthesizer parameters. Thus, by manipulating virtual objects (as taught, for example, by U.S. Pat. 4,988,981, issued Jan. 29, 1991), sounds of different qualities could be created. A skilled user could create modern sounding musical interludes. These ideas have been carried forth by people such as Jaron Lanier who has given a number of public performances in which he manipulates virtual objects to create a musical performance. Research and exploration along these lines is expected to continue (the virtual "air guitar" and the like will probably be developed). In all VR systems of this type, manipulation of a virtual object causes the sound or music to change.
Currently, virtual worlds are created by describing a simulation and a number of objects. The interaction of the objects is described in some form of simulation language or graphical description. Traditionally, the control and creation of the objects is driven by "world building" software. Once a virtual world has been created, a limited number of its parameters may be manipulated by the user from "inside" the virtual world. One example of how these databases are created is described in PCT International Patent Application WO 92/09948, by VPL Research Inc. As is evident from WO 92/09948, it has define animation for all or even some of the virtual objects in a virtual world. Until the present invention, it had not been proposed to interface to nodes in a database defining a virtual environment, and to manipulate such nodes, on the basis of music.
Conventional VR systems and music have thusfar been used together in ways which have the following disadvantages:
(a) a VR system has been used as a virtual musical instrument, so that the user must "play" the virtual instrument (by manipulating an input device) to hear anything. This means that the system creates music, and that the system's musical output is limited by the user's ability to "play" the "instrument;" PA1 (b) VR systems that have given sounds to virtual objects (e.g., the system displays a virtual kitchen sink and produces a "drip-drip" sound which seems to come from the sink's location) have required that the sounds are generated by signals produced within the VR system in response to user manipulation of an input device or internal programs, which signals are then interpreted by a synthesizer. The sounds produced by the synthesizer are thus cued from the VR system in response to manipulation of an input device (which manipulation may, for example, to cause a user to "move" into a position to view or otherwise interact with a virtual kitchen sink from which sounds will then seem to emanate). Thus, these VR systems have depended on user manipulation of an input device to control the appearance or activities of objects in a virtual environment, to cause the VR system to cue production of sound events; and PA1 (c) VR systems have played musical scores as background music for the virtual environment. PA1 (a) an entire song can be choreographed and prerecorded with a control track (for example, indicative of placement and rhythm of dancers), so that the control track forms part of the prerecorded choreographed musical work; PA1 (b) the control track can include higher level information, such as pictures of a dancer or other performer, which can be used as source data by the VR system to display images of the performer in the virtual environment; PA1 (c) the medium for the control track need not the same as that of the music. For example, the music may be recorded on a compact disk (CD) while the control track is recorded on a computer game cartridge or other medium; PA1 (d) synchronization of the control track and the music can be accomplished under control of the VR system, which could use the control track to synchronize with the music, or vice versa; PA1 (e) the control track can be encoded (or processed) in a way which accounts for the "delay time" required for the VR system to use the information coming from the control track. This will improve the apparent synchronization between the music and the graphics data output from the VR system, even when the VR system requires a long time to "draw" a particular frame of an animated virtual world; and PA1 (f) a prerecorded control track can eliminate the need for some embodiments of the invention to include means for automatically decoding musical expression (the automatic decoding of musical expression is poorly understood). PA1 to provide an apparatus which extracts information from music (or other audio) for the control and manipulation of objects within a virtual environment; PA1 to provide an apparatus which uses a control track prerecorded along with audio (music, in preferred embodiments) for the control and manipulation of objects within a virtual environment; PA1 to provide a VR system which delays audio (in response to which control signals are generated) in order to compensate for the lag introduced by other components of the VR system; PA1 to provide a virtual experience in which music effectively drives the display of an animated graphical scene; PA1 to provide a mechanism by which music is used to control and influence a virtual environment in such a way as to relieve the database which describes the virtual environment from having to define all the motions of the objects in the virtual environment; PA1 to provide a control track for the influence and control of a virtual environment in which the control track is created during or following the music recording and production process when individual tracks (of a multi-track musical work) that are used for a particular mix are available before being mixed down; and PA1 to provide a control track which can contain information (such as images of a performer's face, for example) other than information extracted from corresponding music;
Basically, the paradigm to date has been to create systems that have (virtual) object-driven sounds. This invention reverses the paradigm to create a system which has musically-driven objects.
One VR system has been developed in which a VR processor is programmed to perform simple operations to modify a virtual environment in response to voice commands. This VR system, developed at the NASA Ames View Lab during the years 1988-1989, was capable of displaying a virtual object, or terminating the display of a virtual object, in response to a voice command from a human user. However, the system did not produce, modify, or otherwise control a virtual environment in response to music, or in response to a prerecorded control track corresponding to an audio signal.
Outside the VR field, many attempts have been made to produce devices which provide users with visual light effects based on an audio signal, such as music. However, these systems have been disappointing to watch (principally because the light shows are two-dimensional and are not obviously correlated with the audio input), and have typically met with disappointment when marketed.
An example of a conventional apparatus for producing visual light effects based on audio signals is described in U.S. Pat. No. 4,081,829 (issued Mar. 28, 1978). This apparatus controls the display of two-dimensional rings or solid shapes on the screen of a television receiver, in response to audio input signals. However, only a limited set of two-dimensional shapes can be displayed and only limited changes in their shape or color can be accomplished in response to the audio input.
Another example of a conventional apparatus for producing visual light effects in response to audio signals is described in U.S. Pat. No. 4,257,062 (issued Mar. 17, 1981). This apparatus controls a set of lamps which are mounted in eyewear to be worn by the user, by switching individual ones of the lamps on and off in response to music. Peak levels of specific frequency bands of the music are detected and employed to switch on or off different ones of the lamps.
Another system for producing visual effects in response to audio signals has been described in the Jan. 1993 issue of NewMedia magazine (at page 18) as a system which includes a Silicon Graphics Iris Indigo workstation, and which alters the appearance of colored visual representations of sound waves (displayed on a large screen in a concert hall) in response to crowd noise (picked up by a microphone during a concert) and live music in MIDI format (generated by musicians during the concert) supplied to the workstation.
It is believed that prerecorded control tracks (which correspond to prerecorded audio such as music) have not been employed to control operation of a computer system, such as to control generation of a virtual environment by a VR computer system. It is also believed that control signals have not been extracted from music for use in controlling generation of a virtual environment by a VR system (e.g., by populating the virtual environment with animated virtual objects which move in response to the music).